Media object query submission and response

ABSTRACT

Methods and systems for submitting media object queries and receiving suggested answers for the media object queries. In one aspect, a method includes receiving from a first user a first media object and a first query relating to content in the first media object, presenting the first media object and the first query to multiple second users, receiving a suggested answer to the first query from each of two or more second users of the multiple second users, where at least two of the suggested answers are distinct, ranking the suggested answers, and presenting one or more of the ranked suggested answers to the first user.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of, and claims priorityto, U.S. patent application Ser. No. 13/552,587, for Media Object QuerySubmission and Response, to Gaurav Garg and Abhishek Bapna, which wasfiled on Jul. 18, 2012, which is a continuation application of U.S.patent application Ser. No. 12/060,000, for Media Object QuerySubmission and Response, to Gaurav Garg and Abhishek Bapna, which wasfiled on Mar. 31, 2008. The disclosure of the foregoing applications areincorporated here by reference.

BACKGROUND

This specification relates to submitting media object queries andreceiving suggested answers for the media object queries.

Generally, internet searching is based on text queries. Internet searchengines aim to identify resources (e.g., web pages, images, textdocuments, processes, multimedia content) that are relevant to a user'sneeds and to present information about the resources in a manner that isuseful to the user. In response to a text query submitted by a user,search engines return search results referring to resources identifiedas satisfying or matching the text query. Occasionally, people observesomething new around them and are curious to learn more about the newthing. However, if a person cannot identify or describe a particularobject, the person can find it difficult to formulate a meaningful textquery about the object.

SUMMARY

This specification describes technologies relating to submitting mediaobject queries and receiving suggested answers for the media objectqueries.

In general, one aspect of the subject matter described in thisspecification can be embodied in methods that include the actions ofreceiving from a first user a first media object and a first queryrelating to content in the first media object, presenting the firstmedia object and the first query to multiple second users, receiving asuggested answer to the first query from each of two or more secondusers of the multiple second users, where at least two of the suggestedanswers are distinct, ranking the suggested answers, and presenting oneor more of the ranked suggested answers to the first user. Embodimentsof the aspect can include systems, apparatuses, and computer programproducts.

Implementations of the aspect can optionally include one or more of thefollowing features. A suggested answer from a particular second user canbe either a new suggested answer submitted by the particular second userin response to the first query or a previous suggested answer to thefirst query selected by the particular second user, where the previoussuggested answer is a suggested answer previously submitted by adifferent second user of the multiple second users in response to thefirst query. Ranking the suggested answers can further include groupingsimilar suggested answers, for each group of similar suggested answers,determining a rank score for the group by counting the instances that asuggested answer in the group was submitted or selected by a seconduser, for each distinct suggested answer, determining a rank score forthe distinct suggested answer by counting the instances that thedistinct suggested answer was submitted or selected by a second user,and ranking the distinct suggested answers and groups of similarsuggested answers based on the determined rank scores.

A suggested answer from a particular second user can be either a newsuggested answer submitted by the particular second user in response tothe first query or a suggested answer selected by the particular seconduser from multiple suggested answers derived from previous suggestedanswers previously submitted by different second users of the multiplesecond users in response to the first query. Presenting the first mediaobject and the first query to the multiple second users can furtherinclude including the first media object and query pair in a list ofsecond media object and query pairs, the first media object and querypair having a position in the list, and selecting the first media objectand query pair from the list for presentation to the multiple secondusers, where the selection is based on the position of the first mediaobject and query pair in the list.

An indication of interest in the first media object and query pair fromeach of one or more second users of the multiple second users can bereceived. The position of the first media object and query pair in thelist can be modified based on the indications of interest in the firstmedia object and query pair from the one or more second users of themultiple second users, and the first media object and query pair can beselected from the list for presentation to one or more third users,where the selection is based on the modified position of the first mediaobject and query pair in the list. Modifying the position of the firstmedia object and query pair in the list can also be based on the numberof suggested answers submitted or selected by second users of themultiple second users.

Data can be presented to the first user, where the data relates to oneor more media object and query pairs received from the first user or oneor more suggested answers submitted or selected by the first user inresponse to media object and query pairs received from second users. Thesuggested answers to the first query can be received over a specifiedtime period. The media object can be one of an image file, a video file,or an audio file.

In general, in one aspect, a system is provided. The system includes auser interface device, and one or more computers operable to interactwith the user interface device and to: receive from a first user of theuser interface device a first media object and a first query relating tocontent in the first media object, present the first media object andthe first query to multiple second users, receive a suggested answer tothe first query from each of two or more second users of the multiplesecond users, where at least two of the suggested answers are distinct,rank the suggested answers, and present one or more of the rankedsuggested answers to the first user on the user interface device.

Implementations of the aspect can optionally include one or more of thefollowing features. The one or more computers can include a serveroperable to interact with the user interface device through a datacommunication network, and the user interface device can be operable tointeract with the server as a client. The user interface device caninclude a networking interface. The one or more computers can includeone personal computer, and the personal computer can include the userinterface device.

Particular embodiments of the subject matter described in thisspecification can be implemented to realize one or more of the followingadvantages. A community-based platform that allows media object querysubmission and response benefits from the fact that the ability toidentify key features in a media object (e.g., an image or a video) isgreater in humans than in currently available machine algorithms.Aggregating knowledge across multiple users can help answer questionsabout the contents of a media object that would be hard for machines toanswer. The community-based platform also benefits from the fact thatusers are willing to contribute their time and effort to answer queriesfor multiple reasons, e.g., for entertainment, for peer recognition, orbecause of altruistic tendencies.

The details of one or more embodiments of the subject matter describedin this specification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages of thesubject matter will become apparent from the description, the drawings,and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example process for submitting picture and querypairs and receiving suggested answers for the picture and query pairs.

FIG. 2 illustrates an example interface for providing suggested answersfor media object and query pairs.

FIG. 3 illustrates an example interface for providing information to auser about media object and query pairs submitted by the user.

FIG. 4 illustrates an example interface for providing data to a userabout media object and query pairs submitted by the user and suggestedanswers provided by the user.

FIG. 5 shows an example process for receiving media object queries andpresenting ranked suggested answers for the media object queries.

FIG. 6 is a schematic diagram of an example computer system.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

FIG. 1 illustrates an example process 100 for submitting picture andquery pairs and receiving suggested answers for the picture and querypairs. For convenience, the example process 100 will be described withreference to a system (e.g., a community-based platform) that performsthe process 100. Generally, the process 100 receives a picture andrelated query from a user, collects answers provided by other users, andpresents the answers to the submitting user. Although the media objectssubmitted are pictures (e.g., image files) in the example process 100,other types of media objects (e.g., video files or audio files) cansimilarly be submitted by users.

FIG. 1 illustrates a scenario where a user wishes to learn what apicture of Kokopelli represents. In the example process 100, User 1 seesa picture of Kokopelli on a keychain but does not know what it is (110a). User 1 takes a picture of the Kokopelli keychain, for example, usinga camera application on the user's mobile phone (112 a).

User 1 submits the picture of the Kokopelli keychain to a system (e.g.,a community-based platform) and attaches or includes a related textquery like “what on earth is this” (114 a). The system can store thepicture of the Kokopelli keychain and the related text query as the(picture, query) pair_1 (116 a). Similarly, User 2, . . . , User N seeobjects (e.g., people, places, or things) about which they havequestions (110 b, . . . , 110 n) and submit pictures of the objects withrelated text queries to the system, which stores corresponding (picture,query) pair_2, . . . , (picture, query) pair_N (116 b, . . . , 116 n).

A group of users, which can include one or more of User 1, . . . , UserN, view the (picture, query) pairs posted on the system and answer thequeries that catch their interest and for which they have answers (118).For each (picture, query) pair, the system collects the respectiveanswers provided by the contributing users and presents the collectedanswers to the respective user that submitted the (picture, query) pair.In some implementations, an answer is ranked based on how manycontributing users believe that answer to be the correct answer. In someimplementations, the system also captures a general level of interest inthe (picture, query) pair to the community. The system can use the levelof interest to determine whether or not to continue showing the(picture, query) pair to additional users of the system. The system canencourage users to submit queries and to provide answers through areputation mechanism that provides peer recognition, as described below.

In some implementations, a user submits a (picture, query) pair througha web interface (e.g., a web browser running on a personal computer) ora mobile interface (e.g., a wireless application protocol (WAP) browserrunning on a mobile phone). A web interface for a system performing theprocess 100 can include an authentication mechanism to allow the user tolog into the system. For example, if the system is a community-basedplatform, a community member can be required to log into the communityplatform using a user login name and password. In some implementations,the web interface can include the example interfaces 200-400 of FIGS.2-4, which are described in detail below.

A mobile interface for a system performing the process 100 can allow auser to capture a picture of the unknown object and submit the picturewith a text query, e.g., as a Short Message Service (SMS) message or aMultimedia Messaging Service (MMS) message. In some implementations, thephone number of the user's mobile phone is associated with a webinterface login name of the user. A user submitting a picture and querypair on a mobile interface can later view the submitted pair andprovided answers by logging into a web interface for the system.

FIG. 2 illustrates an example interface 200 for providing suggestedanswers for media object and query pairs. The example interface 200 isillustrated as an answer tab 210 of a web interface, e.g., a webinterface of a system that performs the example process 100 of FIG. 1.In some implementations, the web interface can include a question tab(e.g., question tab 310 of FIG. 3) and a karma tab (e.g., karma tab 410of FIG. 4).

The example interface 200 includes a media object 220, a query 230, ananswer area 240, and a level of interest area 250. The media object canbe, for example, an image or a video. In some implementations, theexample interface 200 can include information about the media object,e.g., a date the media object was uploaded to the system, a locationwhere the data of the media object was captured, or the name of the userwho submitted the media object. For example, if the media object isgenerated from data captured from a mobile phone enabled for the GlobalPositioning System (GPS), the GPS coordinates of the mobile phone at thetime the media data was captured can be embedded as metadata in themedia object. The example interface 200 can include the GPS coordinatesas additional information about the media object. The example interface200 can also or alternatively include a hyperlink to a map application,where user selection of the hyperlink triggers the system to present(e.g., in an overlay or a separate interface) a map of the location ofthe GPS coordinates.

In some implementations, an overall quality measure (e.g., a “guru”label) is presented next to the submitting user's name on the exampleinterface 200, indicating the quality of media objects submitted andanswers provided by the submitting user. The query 230 is related tocontent in the media object 220. The query 230 can be directed to aparticular thing in the media object 220. For example, if the mediaobject 220 is a video clip of views from San Francisco, the query 230can ask what the name of the red bridge is. In the example of FIG. 2, auser has submitted an image of a building with a query about where thebuilding is located. In some implementations, the system selects aparticular media object 220 and related query 230 pair for impressingupon a user (i.e., presentation to the user) according to an impressionshare mechanism, described in more detail below.

The answer area 240 includes suggested answers 242, selection elements244 (e.g., checkboxes or buttons), a hyperlink for more results 246, anda text box 248. A user who receives an impression of the media object220 and query 230 pair can respond to the pair by selecting a suggestedanswer 242 using an associated selection element 244. The user can viewadditional suggested answers by selecting the hyperlink for more results246. Alternatively, the user can respond by submitting a new answer,e.g., by inputting the new answer in the text box 248. In someimplementations, the contents of the text box 248 are labeled as the newanswer provided by the particular user.

In some implementations, the suggested answers 242 are answerspreviously submitted by different users in response to viewing the mediaobject 220 and query 230 pair. In other implementations, the suggestedanswers 242 are answers derived from previously suggested answerssubmitted by different users in response to viewing the media object 220and query 230 pair.

In some implementations, the answer area 240 displays the suggestedanswers 242 in order according to how many contributing users selecteach particular suggested answer 242. In other implementations, theanswer area 240 displays the suggested answers 242 in random order toprevent the order from biasing user selection. For example, a userviewing the media object 220 and query 230 pair might be prone to selectthe first suggested answer 242 on the list, e.g., because the user mightbelieve the suggested answers 242 are ranked by popularity and hence,the user might believe that the first suggested answer 242 is likely thecorrect answer.

In some implementations, the answer area 240 does not include suggestedanswers 242 or selection elements 244. For example, for the first userwho views the media object 220 and query 230 pair, the system has notyet received any answers responsive to the media object 220 and query230 pair. The answer area 240 can initially display only the text box248 for receiving answers until at least one responsive answer has beenprovided by a contributing user. In some implementations, the answerarea 240 does not include suggested answers 242 even after responsiveanswers have been received by the system. For example, to prevent anybias that might occur if suggested answers were displayed, the systemcan allow a viewing user to provide an answer only by submitting a newanswer, e.g., through the text box 248, and not display any suggestedanswers 242 for selection.

The level of interest area 250 includes selection elements 252 (e.g.,checkboxes or buttons). Each user viewing the media object 220 and query230 pair can indicate the user's interest in the media object 220 andquery 230 pair using the selection elements 252. In the example of FIG.2, the user can indicate that the user finds the pair interesting, theuser finds the pair not interesting, or the user does not have acomment. In some implementations, the level of interest indicated by theusers (e.g., users of a community-based platform) is used in determiningwhether the media object 220 and query 230 pair is presented toadditional users.

FIG. 3 illustrates an example interface 300 for providing information toa user about media object and query pairs submitted by the user. Theexample interface 300 is illustrated as a question tab 310 of a webinterface, e.g., a web interface of a system that performs the exampleprocess 100 of FIG. 1. The example interface 300 is specific to aparticular user (e.g., a member of a community-based platform with theuser name “John”). In some implementations, the example interface 300allows a user to upload a media object and formulate an associated textquery (e.g., using a text box) relating to content in the media object.The uploaded media object and query pair is added to an archive of mediaobject and query pairs submitted by the user.

The example interface 300 includes a media object 320, a query 330, ananswer area 340, and a user question area 360. In some implementations,the example interface 300 includes information about the media object,e.g., a date the media object was uploaded to the system or the name ofthe user who submitted the media object. In the example of FIG. 3, auser has submitted the picture of Kokopelli with a query asking “what onearth is this.”

The answer area 340 includes suggested answers 342, votes 345, and ahyperlink for more results 346. The suggested answers 342 are rankedaccording to the corresponding votes 345. The votes 345 for a particularsuggested answer 342 indicate how many users agreed with the particularsuggested answer 342. In some implementations, the system groupstogether similar suggested answers (e.g., answers which are permutationsor synonyms of each other, as determined by related words tables) andcombines the votes of the similar suggested answers before ranking. Theuser can view additional suggested answers by selecting the hyperlinkfor more results 346.

In some implementations, the system indicates which suggested answer 342is likely the reasonable answer for the media object 320 and query 330pair. For example, the system can select the suggested answer 342 withthe highest rank (i.e., greatest number of votes 345) as the reasonableanswer. Alternatively, the system can select the reasonable answerbased, in part, on extrinsic information (e.g., information gathered byperforming a search in an image or video repository).

The user question area 360 provides information to the user about othermedia object and query pairs submitted by the user. The user questionarea 360 includes the number of media object and query pairs 364submitted by the user and stored by the system in an archive. The usercan view the other media object and query pairs previously submitted bythe user and the associated suggested answers by selecting a hyperlinkfor previous pairs 366.

FIG. 4 illustrates an example interface 400 for providing data to a userabout media object and query pairs submitted by the user and suggestedanswers provided by the user. The example interface 400 is illustratedas a karma tab 410 of a web interface, e.g., a web interface of a systemthat performs the example process 100 of FIG. 1. The example interface400 is specific to a particular user (e.g., a member of acommunity-based platform with the user name “John”). In someimplementations, the example interface 400 is part of a user profilepage for the particular user as a member of a community.

The example interface 400 includes a user information area 415, aquestion data area 430, an answer data area 440, and a user questionarea 360. The user information area 415 includes information about theuser stored on the system, e.g., a date indicating how long the user hasbeen a member of the community. Similar to FIG. 3, the user questionarea 360 provides information to the user about media object and querypairs previously submitted by the user.

The question data area 430 includes a question quality indicator element432, question quality factors 434, and question quality scores 436. Thequestion quality indicator element 432 indicates an overall qualitylevel for the questions (i.e., media object and query pairs) submittedby the user, as measured by the question quality factors 434. Examplesof question quality factors 434 include an average difficulty level ofquestions, an interest level of the questions, how many users where ableto answer the questions, and how curious is the user. For each questionquality factor 434, there is an associated question quality score 436.The question quality scores 436 can be computed based on user responseto media object and query pairs submitted by the particular user. Forexample, an interest level of the questions can be computed by countingthe user responses provided in the level of interest area 250 of FIG. 2.

The answer data area 440 includes an answer quality indicator element442, answer quality factors 444, and answer quality scores 446. Theanswer quality indicator element 442 indicates an overall quality levelfor the answers submitted by the user, as measured by the answer qualityfactors 444. Examples of answer quality factors 444 include how manydifficult questions were answered by the user and how many of the user'sanswers were the same as or similar to the reasonable answer. There isan answer quality score 446 associated with each answer quality factor444. The answer quality scores 446 can be computed based on theparticular user's responses to media object and query pairs presented tothe particular user. In some implementations, an overall quality measurecan be associated with each user. For example, a user who consistentlysubmits high quality media object and query pairs and consistentlyprovides high quality answers can receive a “guru” label, which can bedisplayed on the user's profile page or next to the user's uploadedmedia objects.

In some implementations, the karma tab 410 of the web interface includesa user interface element (not shown) that allows a user to viewinformation about the users with the highest karma ratings (e.g.,question quality scores 436 and answer quality scores 446). For example,selection by a user of a top karma hyperlink can trigger the system topresent (e.g., in an overlay or in a separate interface) a list of theuser names or user profiles of these highest scoring users.

FIG. 5 shows an example process 500 for receiving media object queriesand presenting ranked suggested answers for the media object queries.For convenience, the example process 500 will be described withreference to FIGS. 2-4 and a system (e.g., a community-based platform)that performs the process 500.

The system receives from a first user a first media object and a firstquery relating to content in the first media object (step 510). Thefirst media object and query pair can be received, for example, in a webinterface similar to the example interface 300 of FIG. 3. Examples ofmedia objects include image files, video files, and audio files. Thefirst media object can be generated by, for example, a device with amicrophone or a camera. A device (e.g., a mobile phone) which has mediadata capturing capability and a network interface (e.g., for wired orwireless communication) can be used to both generate a media object withcaptured media data and submit the media object and query pair to thesystem.

The system presents (i.e., provides impressions of) the first mediaobject and the first query to multiple second users (step 520). In someimplementations, the second users are members of a subgroup (e.g., afriends group or a family group) associated with the first user, wherethe subgroup is part of a larger community, e.g., a social network. Thatis, the initial impressions of the first media object and query pair canbe presented to users known to the first user.

In some implementations, the system includes the first media object andquery pair in a list of second media object and query pairs, where thefirst media object and query pair has a position in the list. The systemselects the first media object and query pair from the list forpresentation to the multiple second users based on the position of thefirst media object and query pair in the list. In some implementations,the system selects the media object and query pair at the top of thelist for presentation to a second user. In some implementations, a mediaobject and query pair will not be presented more than once to the samesecond user. That is, if the media object and query pair at the top ofthe list has already been impressed upon a particular second user, thesecond user will be presented with the next media object and query pairon the list which has not been previously presented to the second user.

In one example, the list of media object and query pairs is part of animpression share mechanism. The media object and query pairs are orderedin the list according to each pair's impression score, where ordering isfrom highest to lowest impression score. Each new media object and querypair submitted to the system can initially receive a maximum impressionscore (e.g., 500), thereby positioning the new media object and querypair at the top of the list. During an initial impression period, thenew media object and query pair receives a default number ofimpressions. For example, a new media object and query pair can bepresented to second users a defined number of times (e.g., 100) bydefault in the initial impression period. After the initial impressionperiod, the system can determine whether the particular media object andquery pair will be presented to additional users based on the secondusers' responses, which modify the pair's impression score as describedin more detail below.

The system receives a suggested answer to the first media object and thefirst query from each of two or more second users, where at least two ofthe suggested answers are distinct (step 530). A suggested answer from aparticular second user can be a new suggested answer submitted by theparticular second user in response to the first media object and querypair, or the suggested answer can be a suggested answer previouslysubmitted by a different second user in response to the first mediaobject and query pair. Alternatively, a suggested answer can be derivedfrom previous suggested answers previously submitted by different secondusers in response to the first media object and query pair. For example,a derived suggested answer can be a previously submitted suggestedanswer with the spelling corrected. In some implementations, thesuggested answers to the first media object and query pair are receivedover a specified time period. In other implementations, the suggestedanswers to the first media object and query pair are received over anunspecified time period. For example, the suggested answers can bereceived over the time period required to provide the default number ofimpressions of the first media object and query pair to second users.

The system ranks the suggested answers (step 540). In someimplementations, the system groups similar suggested answers beforeranking. The suggested answers can be grouped together, for example,based on identifying the semantic similarity between the suggestedanswers. In one example, the system subdivides a suggested answer intosmaller units of words or phrases (e.g., n-grams of n numbers of words)and compares the units with units from another suggested answer, e.g.,by searching for the units in related words tables. The comparison cangenerate a semantic similarity score between the two suggested answers.If the semantic similarity score is greater than a defined thresholdscore, the system can determine that the two suggested answers aresufficiently similar to be grouped together before ranking.

For each group of similar suggested answers, the system can determine arank score for the group by counting the instances that a suggestedanswer in the group was provided (e.g., submitted or selected) by asecond user. For each distinct suggested answer, the system candetermine a rank score for the distinct suggested answer by counting theinstances that the distinct suggested answer was provided by a seconduser. The system can rank the distinct suggested answers and groups ofsimilar suggested answers based on the determined rank scores.

The system presents one or more of the ranked suggested answers to thefirst user (step 550). In some implementations, the system receives anindication of interest in the first media object and query pair fromeach of one or more second users presented with the pair. Theindications of interest can be received, for example, in a web interfacesimilar to the example interface 200 of FIG. 2. The system can modifythe position of the first media object and query pair in the list ofmedia object and query pairs based on the indications of interest in thefirst media object and query pair. In some implementations, the pair'sposition in the list is also modified based on the number of suggestedanswers provide (e.g., submitted or selected) by the second users. Thesystem can select the first media object and query pair from the listfor presentation as an impression to one or more third users based onthe modified position of the first media object and query pair in thelist.

In one example, the system modifies the position of the first mediaobject and query pair in the list by updating the pair's impressionscore based on the second users' responses (e.g., the second users'level of interest in the pair and the number of answers provided by thesecond users). Table 1 illustrates example impression scoring weightsfor modifying a media object and query pair's impression score.

TABLE 1 Example impression scoring weights Not No Response InterestingComment Interesting Answer −5 −1 1 No Answer −2 2 5Table 1 divides user responses to a particular media object and querypair into six categories with an associated weight for each category.For each second user receiving an impression of the first media objectand query pair, the system increments a counter for the respectiveresponse category. For example, if a second user indicates that thefirst media object and query pair is not interesting (e.g., using therespective selection element 252 of FIG. 2) but does not provide asuggested answer, the system increments a counter for the (No Answer,Not Interesting) category with an associated weight of −2.

At the end of the initial impression period in which the first mediaobject and query pair had the maximum impression score (e.g., 500), thesystem weights the counter value for each category by the associatedweight and sums the weighted values to determine an updated impressionscore for the pair. For example, if 8, 12, and 25 second users providedanswers and indicated the pair was not interesting, provided no comment,and indicated the pair was interesting, respectively, and 15, 18, and 22users did not provide answers and indicated the pair was notinteresting, provided no comment, and indicated the pair wasinteresting, respectively, the updated impression score would becomputed as (8*−5)+(12*−1)+(25*1)+(15*−2)+(18*2)+(22*5)=89. After theinitial impression period, the position of the first media object andquery pair in the list of pairs is modified based on the updatedimpression score.

If a particular media object and query pair has a high updatedimpression score (e.g., because most second users thought the pair wasinteresting but did not provide answers), the media object and querypair will remain high on the list but will be positioned below the newpairs with the maximum impression score. In contrast, if a particularmedia object and query pair has a low updated impression score (e.g.,because most second users thought the pair was not interesting andprovided answers), the media object and query pair will drop low on thelist, and the likelihood that the system will select the pair forpresentation to third users is greatly reduced.

In some implementations, the system removes a media object and querypair from the list if the updated impression score is below a determinedthreshold. In some implementations, following the initial impressionperiod, the system updates the impression score of a media object andquery pair on a scheduled basis (e.g., every additional 100 impressionsor every week). In some implementations, the system removes a mediaobject and query pair from the list after the media object and querypair has been on the list for a determined period of time. In otherimplementations, the system removes a media object and query pair fromthe list after the media object and query pair has had a maximum numberof impressions or has received a maximum number of responses.

In some implementations, the system presents data to the first userrelating to one or more media object and query pairs received from thefirst user or one or more suggested answers provided by the first userin response to media object and query pairs received from second users.For example, the system can provide to the first user (e.g., on a userprofile page of a community-based platform) statistics about the qualityof the media object and query pairs submitted by the user and statisticsabout the quality of the suggested answers provided by the user. Theseusers statistics can be presented to the first user, for example, in aweb interface similar to the example interface 400 of FIG. 4.

In some implementations, other users see some or all of the first user'sstatistics. In some implementations, other users see an overall qualitymeasure (e.g., stars or labels) associated with the first user, forexample, next to media object and query pairs submitted by the firstuser. Allowing users to view other users' overall quality measures orstatistics provides a way to recognize and encourage high performingusers of the system.

In some implementations, the system divides media object and query pairsinto categories, e.g., people, places, and things. The system canmaintain separate lists of media object and query pairs for eachcategory. A user can specify a particular category for submitting amedia object and query pair and can request to receive impressions ofmedia object and query pairs from a particular category. In someimplementations, the system automatically determines how a submittedmedia object and query pair should be categorized (e.g., using imageprocessing algorithms).

In some implementations, the system recognizes submitted images or videoas pornography, e.g., by scanning the image or video for skin color. Thesystem can then reject the media object or can automatically categorizethe media object as adult material. In some implementations, the systemcan operate in a safe search mode, which blocks adult material and othermature content from being presented to a user.

In some implementations, the system includes a translation service. Forexample, the system can accommodate a language preference associatedwith a user by presenting queries and suggested answers translated tothe user's preferred language, e.g., using statistical machinetranslation or a translation dictionary. A language preference can beassociated with a user when, for example, the user specifies thelanguage preference or the system infers a language preference, e.g.,from the language used by the user or the domain accessed.

FIG. 6 is a schematic diagram of an example computer system 600. Thesystem 600 can be used for performing the actions and methods describedabove. The system 600 can include a processor 618, a memory 616, astorage device 652, and input/output devices 654. Each of the components618, 616, 652, and 654 are interconnected using a system bus 656. Theprocessor 618 is capable of processing instructions within the system600. These instructions can implement one or more aspects of thesystems, components, and techniques described above. In someimplementations, the processor 618 is a single-threaded processor. Inother implementations, the processor 618 is a multi-threaded processor.The processor 618 can include multiple processing cores and is capableof processing instructions stored in the memory 616 or on the storagedevice 652 to display graphical information for a user interface on theinput/output device 654.

The memory 616 is a computer readable medium such as volatile ornon-volatile that stores information within the system 600. The memory616 can store processes related to the functionality of the searchengine 1030, for example. The storage device 652 is capable of providingpersistent storage for the system 600. The storage device 652 caninclude a floppy disk device, a hard disk device, an optical diskdevice, or a tape device, or other suitable persistent storage mediums.The storage device 652 can store the various databases described above.The input/output device 654 provides input/output operations for thesystem 600. The input/output device 654 can include a keyboard, apointing device, and a display unit for displaying graphical userinterfaces.

The computer system shown in FIG. 6 is but one example. In general,embodiments of the subject matter and the functional operationsdescribed in this specification can be implemented in digital electroniccircuitry, or in computer software, firmware, or hardware, including thestructures disclosed in this specification and their structuralequivalents, or in combinations of one or more of them. Embodiments ofthe subject matter described in this specification can be implemented asone or more computer program products, i.e., one or more modules ofcomputer program instructions encoded on a tangible program carrier forexecution by, or to control the operation of, data processing apparatus.The tangible program carrier can be a propagated signal or a computerreadable medium. The propagated signal is an artificially generatedsignal, e.g., a machine generated electrical, optical, orelectromagnetic signal, that is generated to encode information fortransmission to suitable receiver apparatus for execution by a computer.The computer readable medium can be a machine readable storage device, amachine readable storage substrate, a memory device, a composition ofmatter effecting a machine readable propagated signal, or a combinationof one or more of them.

The term “data processing apparatus” encompasses all apparatus, devices,and machines for processing data, including by way of example aprogrammable processor, a computer, or multiple processors or computers.The apparatus can include, in addition to hardware, code that creates anexecution environment for the computer program in question, e.g., codethat constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, or a combination of one or moreof them.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, or declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, or other unitsuitable for use in a computing environment. A computer program does notnecessarily correspond to a file in a file system. A program can bestored in a portion of a file that holds other programs or data (e.g.,one or more scripts stored in a markup language document), in a singlefile dedicated to the program in question, or in multiple coordinatedfiles (e.g., files that store one or more modules, sub-programs, orportions of code). A computer program can be deployed to be executed onone computer or on multiple computers that are located at one site ordistributed across multiple sites and interconnected by a communicationnetwork.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. The essential elements of a computer area processor for performing instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto-optical disks, or optical disks. However, a computerneed not have such devices. Moreover, a computer can be embedded inanother device, e.g., a mobile telephone, a personal digital assistant(PDA), a mobile audio or video player, a game console, a GlobalPositioning System (GPS) receiver, to name just a few.

Computer-readable media suitable for storing computer programinstructions and data include all forms of non-volatile memory, mediaand memory devices, including by way of example semiconductor memorydevices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks,e.g., internal hard disks or removable disks; magneto-optical disks; andCD-ROM and DVD-ROM disks. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back-end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front-end component, e.g., aclient computer having a graphical user interface or a Web browserthrough which a user can interact with an implementation of the subjectmatter described is this specification, or any combination of one ormore such back-end, middleware, or front-end components. The componentsof the system can be interconnected by any form or medium of digitaldata communication, e.g., a communication network. Examples ofcommunication networks include a local area network (“LAN”) and a widearea network (“WAN”), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinvention or of what may be claimed, but rather as descriptions offeatures that may be specific to particular embodiments of particularinventions. Certain features that are described in this specification inthe context of separate embodiments can also be implemented incombination in a single embodiment. Conversely, various features thatare described in the context of a single embodiment can also beimplemented in multiple embodiments separately or in any suitablesubcombination. Moreover, although features may be described above asacting in certain combinations and even initially claimed as such, oneor more features from a claimed combination can in some cases be excisedfrom the combination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated together in a single software product orpackaged into multiple software products.

Particular embodiments of the subject matter described in thisspecification have been described. Other embodiments are within thescope of the following claims. For example, the actions recited in theclaims can be performed in a different order and still achieve desirableresults. As one example, the processes depicted in the accompanyingfigures do not necessarily require the particular order shown, orsequential order, to achieve desirable results. In certainimplementations, multitasking and parallel processing may beadvantageous.

What is claimed is:
 1. A method comprising: receiving, from a userdevice of a first user, an image and a text query related to the image,wherein the text query requests information identifying contentpresented in the image; providing for presentation the image and thetext query to a plurality of second users different from the first user;and receiving a suggested answer to the text query from each of a groupof second users of the plurality of second users, and wherein asuggested answer from a particular second user is either a new suggestedanswer submitted by the particular second user in response to the textquery or a previous suggested answer to the text query, the previoussuggested answer being a suggested answer previously submitted by adifferent second user of the plurality of second users in response tothe text query.
 2. The method of claim 1, further comprising: providingone or more of the suggested answer for presentation to the first user.3. The method of claim 2, wherein providing one or more of the suggestedanswers includes providing one or more ranked suggested answers andwherein providing one or more ranked suggested answers comprises:grouping similar suggested answers based on a semantic similaritymeasure; and ranking the suggested answers where each group of similarsuggested answers is given a combined ranking.
 4. The method of claim 3,wherein ranking the suggested answers further comprises: for each groupof similar suggested answers, determining a rank score for the group bycounting the instances that a suggested answer in the group wassubmitted or selected by a second user; for each distinct suggestedanswer, determining a rank score for the distinct suggested answer bycounting the instances that the distinct suggested answer was submittedor selected by a second user; and ranking the distinct suggested answersand groups of similar suggested answers based on the determined rankscores.
 5. The method of claim 1, wherein providing the image and thetext query to the plurality of second users further comprises: includingthe image and the text query pair in a list of additional image andquery pairs, the image and text query pair having a position in thelist; and selecting the image and text query pair from the list forpresentation to the plurality of second users, the selection based onthe position of the image and text query pair in the list.
 6. The methodof claim 5, further comprising: modifying a position of the image andtext query pair in the list based on indications of interest in theimage and text query pair from one or more second users of the pluralityof second users; and selecting the image and text query pair from thelist for presentation to one or more third users, the selection based onthe modified position of the image and text query pair in the list. 7.The method of claim 5, further comprising: determining an impressionscore for the image and text query pair based on an indication ofinterest received from each second user; and modifying a position of theimage and the text query pair in the list provided to a plurality ofthird users based on the impression score.
 8. The method of claim 1,comprising: providing data to the first user for presentation, the datarelating to one or more image and text query pairs received from thefirst user or one or more suggested answers submitted or selected by thefirst user in response to image and text query pairs received from oneor more second users.
 9. A system comprising: one or more computers andone or more storage devices storing instructions that are operable, whenexecuted by the one or more computers, to cause the one or morecomputers to perform operations comprising: receiving, from a userdevice of a first user, an image and a text query related to the image,wherein the text query requests information identifying contentpresented in the image; providing for presentation the image and thetext query to a plurality of second users different from the first user;and receiving a suggested answer to the text query from each of a groupof second users of the plurality of second users, and wherein asuggested answer from a particular second user is either a new suggestedanswer submitted by the particular second user in response to the textquery or a previous suggested answer to the text query, the previoussuggested answer being a suggested answer previously submitted by adifferent second user of the plurality of second users in response tothe text query.
 10. The system of claim 9, wherein the operationsfurther comprise: providing one or more of the suggested answer forpresentation to the first user.
 11. The system of claim 10, whereinproviding one or more of the suggested answers includes providing one ormore ranked suggested answers and wherein providing one or more rankedsuggested answers comprises: grouping similar suggested answers based ona semantic similarity measure; and ranking the suggested answers whereeach group of similar suggested answers is given a combined ranking. 12.The system of claim 11, wherein ranking the suggested answers furthercomprises: for each group of similar suggested answers, determining arank score for the group by counting the instances that a suggestedanswer in the group was submitted or selected by a second user; for eachdistinct suggested answer, determining a rank score for the distinctsuggested answer by counting the instances that the distinct suggestedanswer was submitted or selected by a second user; and ranking thedistinct suggested answers and groups of similar suggested answers basedon the determined rank scores.
 13. The system of claim 9, whereinproviding the image and the text query to the plurality of second usersfurther comprises: including the image and the text query pair in a listof additional image and query pairs, the image and text query pairhaving a position in the list; and selecting the image and text querypair from the list for presentation to the plurality of second users,the selection based on the position of the image and text query pair inthe list.
 14. The system of claim 13, wherein the operations furthercomprise: modifying a position of the image and text query pair in thelist based on indications of interest in the image and text query pairfrom one or more second users of the plurality of second users; andselecting the image and text query pair from the list for presentationto one or more third users, the selection based on the modified positionof the image and text query pair in the list.
 15. The system of claim13, wherein the operations further comprise: determining an impressionscore for the image and text query pair based on an indication ofinterest received from each second user; and modifying a position of theimage and text pair in the list provided to a plurality of third usersbased on the impression score.
 16. The system of claim 9, wherein theoperations comprise: providing data to the first user for presentation,the data relating to one or more image and text query pairs receivedfrom the first user or one or more suggested answers submitted orselected by the first user in response to image and text query pairsreceived from one or more second users.
 17. A non-transitory computerstorage medium encoded with a computer program, the program comprisinginstructions that when executed by one or more computers cause the oneor more computers to perform operations comprising: receiving, from auser device of a first user, an image and a text query related to theimage, wherein the text query requests information identifying contentpresented in the image; providing for presentation the image and thetext query to a plurality of second users different from the first user;and receiving a suggested answer to the text query from each of a groupof second users of the plurality of second users, and wherein asuggested answer from a particular second user is either a new suggestedanswer submitted by the particular second user in response to the textquery or a previous suggested answer to the text query, the previoussuggested answer being a suggested answer previously submitted by adifferent second user of the plurality of second users in response tothe text query.
 18. The computer storage medium of claim 17, furthercomprising instructions that when executed by one or more computerscause the one or more computers to perform operations comprising:providing one or more of the suggested answer for presentation to thefirst user.
 19. The computer storage medium of claim 18, whereinproviding one or more of the suggested answers includes providing one ormore ranked suggested answers and wherein providing one or more rankedsuggested answers comprises: grouping similar suggested answers based ona semantic similarity measure; and ranking the suggested answers whereeach group of similar suggested answers is given a combined ranking. 20.The computer storage medium of claim 17, wherein providing the image andthe text query to the plurality of second users further comprises:including the image and the text query pair in a list of additionalimage and query pairs, the image and text query pair having a positionin the list; and selecting the image and text query pair from the listfor presentation to the plurality of second users, the selection basedon the position of the image and text query pair in the list.